Hospital-wide survey of clinical experience with artificial intelligence applied to daily chest radiographs

Purpose To assess experience with and perceptions of clinical application of artificial intelligence (AI) to chest radiographs among doctors in a single hospital. Materials and methods A hospital-wide online survey of the use of commercially available AI-based lesion detection software for chest radiographs was conducted with all clinicians and radiologists at our hospital in this prospective study. In our hospital, version 2 of the abovementioned software was utilized from March 2020 to February 2021 and could detect three types of lesions. Version 3 was utilized for chest radiographs by detecting nine types of lesions from March 2021. The participants of this survey answered questions on their own experience using AI-based software in daily practice. The questionnaires were composed of single choice, multiple choices, and scale bar questions. Answers were analyzed according to the clinicians and radiologists using paired t-test and the Wilcoxon rank-sum test. Results One hundred twenty-three doctors answered the survey, and 74% completed all questions. The proportion of individuals who utilized AI was higher among radiologists than clinicians (82.5% vs. 45.9%, p = 0.008). AI was perceived as being the most useful in the emergency room, and pneumothorax was considered the most valuable finding. Approximately 21% of clinicians and 16% of radiologists changed their own reading results after referring to AI, and trust levels for AI were 64.9% and 66.5%, respectively. Participants thought AI helped reduce reading times and reading requests. They answered that AI helped increase diagnostic accuracy and were more positive about AI after actual usage. Conclusion Actual adaptation of AI for daily chest radiographs received overall positive feedback from clinicians and radiologists in this hospital-wide survey. Participating doctors preferred to use AI and regarded it more favorably after actual working with the AI-based software in daily clinical practice.

was obtained before they began to answer the 25 questions of the online survey autonomously under the guidance of the IRB of our hospital. The online surveys were uploaded in a web-based format (SurveyMonkey.com). All study methods were in accordance to the Consensus-based Checklist for Reporting of Survey Studies guidelines. All participants were either clinicians or radiologists who worked in our hospital during 2020 or 2021. The complete question forms are provided in S1 File. Participants answered questions concerning their clinical experience with AI-based lesion detection software for chest radiographs. Questions were designed to gather information on basic demographics, experience with AI, actual individual utilization status of AI, and preferences and attitudes toward AI-based software after actual usage. The questionnaires were composed of a single choice, multiple choices, and scale bar questions, and there were no open-ended questions. The composition of questionnaires and way of presenting answers were constructed and validated under the guidance of a statistical expert to analysis results in an objective way as much as possible and to overcome the radical subjectivity of a survey study.

Use of AI-based lesion detection software for chest radiographs
In our general hospital, commercially available AI-based lesion detection software (Lunit INSIGHT CXR, versions 2 and 3, Lunit Inc., Korea.) has been run on all chest radiographs from patients over 18 years old since March 2020. This ResNet34-based software was developed and approved for adult chest radiographs with the anteroposterior and posteroanterior view [23]. Detailed information about the integration process of AI for chest radiographs was well introduced in a recent study [20]. In our hospital, version 2 of the abovementioned software was utilized from March 2020 to February 2021 and could detect three types of lesions (nodule, consolidation, and pneumothorax) (Fig 1A). From March 2021, version 3 was utilized for chest radiographs by detecting nine types of lesions (nodule, consolidation, pneumothorax, pneumoperitoneum, fibrosis, atelectasis, cardiomegaly, calcification, and pleural effusion). This was an upgraded version including three types of the lesions detected in version 2, and these two versions were not from different software. Under the guidance of the radiology department of our hospital, version 2 was used from March 2020 to February 2021, and version 3 replaced version 2 from March 2021 and was used for all chest radiographs since then. Therefore, the users of version 2 and 3 were not different. The abbreviations and abnormality scores of each lesion are displayed with an additional grayscale heatmap at the lesion location in version 3 ( Fig 1B).
The workflow for utilizing the AI system in our picture archiving and communication system (PACS) is shown in Fig 2. As soon as the chest radiographs were verified by the radiographers, the images were automatically sent to the AI processing server. PACS automatically retrieved the AI results and made them accessible on the radiologists' or clinicians' workstations. PACS viewer software (Zetta PACS, Taeyoung soft Co. Ltd., Korea) presented the total abnormality score on a worklist with several display options for the results. Contour maps were attached as separate captured images following the original radiographs. An abnormality score of 0.15 was the cutoff for visualization on the contour maps for each lesion according to the vendors' guidelines and other studies [23][24][25][26]. Through this process, doctors could refer to the analyzed images simply by scrolling down from the original radiographs whenever they wished to refer to the AI results. Therefore, the participants of this survey answered questions on their own experience using AI-based software in daily practice from March 2020.

Statistical analysis
SAS software version 9.4 (SAS Institute Inc., Cary, NC, USA) was used for the statistical analysis. Data are presented as means with standard deviations and medians with interquartile ranges (Q1, Q3). The two-sample t-test or Fisher's exact test was used for group comparison. The paired t-test was used for comparisons within groups, and the Wilcoxon rank-sum test was used for comparison between groups. P-values less than 0.05 were considered as statistically significant.

Demographics
Among 194 doctors in our hospital, a total of 123 doctors (63.4%) answered the survey, and 91 (clinicians: radiologists = 78:13) completed it in full. The mean proportion of completed questions among all given questions was 79%, and the survey took an average 7 minutes, 48 seconds to complete. Basic participant demographics are summarized in Table 1. Most doctors of our hospital were board-certified staff because our general hospital newly opened in March 2020. Except for several interns and residents in family medicine, there were no trainees in most departments, including radiology, during the study period. Thus, board-certified doctors accounted for 83.7% of the survey participants, and the remaining 16.3% were residents or interns. The subspecialties of the participants are displayed in Fig 3. Among the radiologists, there was only one thoracic radiologist. However, in our hospital, all chest radiographs are interpreted by board-certified radiologists regardless of their subspecialty, because the number of chest radiographs that need to be interpreted are high. Radiologists are expected to read a minimum of 500 radiographs every month. The mean number of adult chest radiographs obtained per month from March 2020 to July 2021 was 6,849. The proportion of doctors who had previous experience with AI-based education was significantly smaller among clinicians than radiologists (27.6% vs. 93.3%, p<0.001). The proportion of doctors who had experience with AI-based research was also smaller among clinicians (21% vs. 46.7%, p = 0.049). When asked about overall personal experience with AI (Table 2), clinicians and radiologists showed significant increases in experience after March 2020, compared to before (all, p<0.005), and this increase was more pronounced among radiologists (38% vs. 54.6%, p = 0.01).

Utilization of AI for chest radiographs
As shown in Table 2, the proportions of chest radiographs among all utilized imaging studies in daily practice were 45.2% for clinicians and 18.8% for radiologists (p = 0.003). However, the proportion of chest radiographs for which doctors utilized AI results in a day was significantly higher among radiologists than clinicians (45.9% vs. 82.5%, p = 0.008). When participants were asked to pick the location where they thought the AI results were put to best use, they answered the emergency room (ER), outpatient unit, inpatient unit, and intensive care unit (ICU) in descending order. When radiologists were asked to choose the most useful finding among the nine types of lesions assessed by AI, they answered pneumothorax, nodules, consolidation, atelectasis, pneumoperitoneum, pleural effusion, cardiomegaly, fibrosis, and calcification in descending order (Fig 4). When asked to choose between versions 2 and 3, 83.3% of clinicians and 84.6% of radiologists preferred version 3 of the AI-based software. However, on

Multiple-choice questions assessing AI experience
Participants were allowed to choose multiple answers for each question (Table 3). When asked why they referred to AI results, 74.7% of the participants answered that they referred to AI results with the belief that it would help lower the risk of missed diagnoses. About 35.2% of the participants answered it was because it was easy to refer to the AI results on PACS, 27.5% said it had become routine to check the AI results, and 19.8% said they referred to the AI results for their relative accuracy. When surveying to which results the participants mainly referred, 69.2% of the participants answered the location of the lesion that was displayed on chest radiographs, 49.5% selected the total abnormality score of each radiograph, 46.2% chose per lesion abbreviations, and 25.3% selected per lesion abnormality score. When asked for what each regarded as the most useful advantage of incorporating AI, 65.9% selected lesion detection, 59.3% chose discrimination of normal and abnormal radiographs, 47.3% chose quicker decision making, 13.2% selected differential diagnosis of lesions, and 6.6% selected triage of reading.  When asked for which function or information they wanted to be developed for AI in the future, 59.3% of the participants chose a comparison function that would automatically compare images to previous radiographs, such as the comparison of pleural effusion or pneumothorax. About 53.8% selected increased diagnostic accuracy for lesion detection and also an alarm system for urgent lesions on radiographs. About 30.8% chose expanded use to different types of radiographs other than chest imaging, 18.7% selected broadening the applicable age to patients less than 18 years old, and 17.6% selected increasing the types of lesions detectable on chest radiographs.

Scale-bar questions assessing AI experience
We asked to subjectively choose percentages from 0 to 100% or -50 to +50% using the sliding bar function on the website to compare perceptions of AI before and after use thereof ( Table 4). As many as 21% of clinicians and 16% of radiologists said that they had changed their own reading results after referring to the AI results (p = 0.727). Clinicians and radiologists said their trust levels for AI were about 64.9% and 66.5%, respectively (p = 0.759).
We used the -50 to +50% scales when the answers could be binary, such as a negative or positive attitude, or decreased or increased results according to the participants' perceptions about AI usage. When asked how the AI results would affect the reading times of chest radiographs, clinicians and radiologists said that they thought AI helped to reduce reading times and the number of reading requests for chest radiographs. In comparison of answers between clinicians and radiologists, the only significant difference was observed in the question concerning reading requests. Radiologists thought that reading requests for chest radiographs were reduced more than indicated by clinicians (-23.5% vs. -12.2%, p = 0.026). In addition, participants thought that diagnostic accuracy would increase after using AI, and they were more positive about AI-based medical devices after using them in clinical practice. Both clinicians and radiologists felt positive about the future usage of AI.

Comparison of staff and trainees among clinicians
The overall answers of trainees among clinicians are summarized in S2 File. There were no significant differences in perceptions of AI-based devices between staff and trainees among clinicians.

Discussion
Our study showed that about 46% of clinicians and 83% of radiologists referred to the analyzed results of the AI-based software in daily clinical practice for chest radiographs. The most common reason for using the software was to reduce missed diagnoses, and the second reason was because the software made it easy to utilize the AI results on PACS. Interestingly, 28% of participants answered that referring to the AI results had become routine during their readings of chest radiographs. This shows that presenting analyzed results with an efficient and userfriendly interface is critical for the successful adaptation of AI into the clinical process. It is notable that many doctors now routinely refer to AI in their everyday workflow as this gives us a glimpse of what full adaptation of AI can mean for radiology in the future. The most useful location for utilizing the AI results was the ER and the outpatient unit. This indicates that the AI-based software was more useful in locations that require urgent decisions to be made that cannot wait a radiologist's reading. Among the various lesion types, urgent or important lesions were thought to reap the benefits of AI results, with pneumothorax, nodule, and consolidation in decreasing order, even though the pneumoperitoneum was ranked lower. This result may be influenced by the incidence of detected lesion types to depict the most useful finding by the participants. Our results suggest that AI-based software can be adapted to chest radiographs effectively. This is important because chest radiographs are still one of the first imaging tools used to guide future treatment and that AI can help depict urgent conditions on chest radiographs even in situations where it is difficult to get a reading right away from a radiologist who is already dealing with a large number of daily images [27]. This means AI can help clinicians and radiologists to catch important diseases and can be adapted for critical diseases first and more effectively [27]. Concerning their experience with AI and the buildup of trust in the results from AI, clinicians and radiologists said they had changed their own reading results after referring to AI, with these changes thought to occur in 21% and 16% of cases, respectively. Doctors rated the trust levels for AI results as 64.9% and 66.5% for clinicians and radiologists, respectively. The AI-based software had a tendency to reduce reading times and the number of reading requests for chest radiographs. Concerning the perception for AI between radiologists and clinicians, radiologists thought reading requests for chest radiographs were reduced more than clinicians. This would be because compared to the various number of clinicians who give reading requests, a limited number of radiologists had to read chest radiographs for the entire hospital. This could make radiologists feel the effect of AI more on reducing reading requests after integrating AI. In addition, participants answered that using the software increased diagnostic accuracy and that they regarded AI in a more positive light after actual usage. They were optimistic about the future usage of AI. Another interesting thing to note is that doctors seem to have truly accepted the adaptation of AI-based software for daily chest radiographs based on the overall positive feedback collected through the survey responses. This study was meaningful because it gives a broad picture of the actual clinical effect of AI and how it is perceived through the eyes of a relatively large number of doctors from an entire hospital.
In a recent survey of trainees and fellows including radiologists from two nations, 60% of the participants answered that AI would impact clinical reality in less than 5 years, especially for screening disease and reducing the time needed for monotonous work [15]. However, 80.9% of clinicians answered in the same survey that they did not have any actual experience with AI despite these high expectations [15]. Still, AI-based methods have been developed and validated for various diseases, but mostly in a retrospective manner, and there are not many conclusions based on external validation or genuine clinical use [3,8,19,[28][29][30]. In another large survey in Europe, the degree of AI-based knowledge inversely affected fear about AI and affected perceptions about AI [10]. Another survey by the same research group showed that less knowledge and ethical issues could interrupt the wide adaptation of AI in clinical practice [12]. Despite implementation of AI being in its beginning stages, most studies showed a positive attitude toward AI in radiology and acceptance of its inevitable adaptation to medical imaging [11,15,23]. In a recent consensus statement, expert chest radiologists concluded that AI-based medical devices could help clinicians make decisions when radiologists are not promptly available and that AI could act as an assistant for radiologists [4]. Creating an effective clinical environment is a key factor for the successful adaptation of AI-based devices in medicine [4,20,22,31]. Our study is meaningful because it demonstrates how commercially available AI-based software has actually been implemented in clinical practice and integrated successfully from an entire hospital-wide perspective. Chest radiographs are commonly performed and utilized for all subspecialties. Therefore, integration of AI on chest radiographs could broadly impact clinical practices [24,26,[32][33][34], and our survey demonstrated how doctors are affected after AI is integrated into the daily imaging process in March 2020. In addition, our study showed that the clinicians and radiologists of our institution regarded AI more favorably after working with the AI-based software.
Our study has several limitations. First, according to our hospital's characteristics, we could not compare actual experience before and after the adaptation of AI because our hospital adopted this software from its first opening. In addition, we could not include radiology trainees because most departments of our hospital did not have them at that time and doctors of relatively young age consisted a large proportion of the whole medical team (about 59.4% of participants were in their twenties or thirties), compared to other hospitals. The responses to the survey questions were based on each doctor's personal experience before and after working at our hospital. To obtain qualified and objective results, we asked the participants of the survey to answer using mostly 0-100% or -50-+50% scales for dedicated answers. Second, the actual effect of AI on workflow and outcomes was not assessed quantitatively in this survey study. According to the nature of survey study, this was mainly based on participants' perceptions of how accurate the AI was. However, because the adapted period was not short and there are very few surveys conducted by various doctors of different departments in an entire hospital unit, we thought that the influence of AI on clinical practice would be reflected in this survey and that this study would be of interest to the readers in the recent status of AI for radiology. Because this survey was performed in 2021, the doctors' acceptance of and experience with AI could have changed as time passes. Further continuous studies demonstrating how AI has changed actual workflows and the perception of doctors and has influenced clinical outcomes are needed as experience accumulates. We are in the process of demonstrating the effect of AI on diagnostic accuracy or reading time to justify these survey results in an objective way and hope to confirm our findings with quantitative results in the next step of our research.

Conclusions
The real adaptation of an AI-based software for daily chest radiographs received overall positive feedback from clinicians and radiologists in this hospital-wide survey. They preferred to use AI to reduce missed diagnoses, and the most useful location for utilizing the AI was the ER. The survey participants thought that AI could help them to catch important diseases and be adapted for critical diseases effectively. The clinicians and radiologists regarded AI more favorably after actual working with the AI-based software in daily clinical practice.